DMS: Create deployment and version during `bundle deploy` by shreyas-goenka · Pull Request #5386 · databricks/cli

shreyas-goenka · 2026-05-31T21:28:54Z

We need to track deployment state in the server. This PR just creates the scaffolding version and deployment when a user approves the plan to apply it.

Note: because lineage is persisted in WAL already we minimize the risk of orphaned deployments (first create request fails) becase we always read lineage from local WAL during replay.

eng-dev-ecosystem-bot · 2026-05-31T22:17:38Z

Integration test report

Commit: caeb738

Run: 28368152735

	Env	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🟨	aws linux	7	2	1	13	230	1039	5:50
🟨	aws windows	7		1	13	234	1037	6:35
💚	aws-ucws linux			8	13	316	957	4:48
💚	aws-ucws windows			8	13	318	955	3:59
💚	azure linux			2	15	232	1038	4:00
💚	azure windows			2	15	234	1036	3:47
💚	azure-ucws linux			2	15	318	954	5:12
💚	azure-ucws windows			2	15	320	952	3:50
💚	gcp linux			2	15	231	1040	3:29
💚	gcp windows			2	15	233	1038	3:20

23 interesting tests: 13 SKIP, 7 KNOWN, 2 flaky, 1 RECOVERED

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🙈	TestAccept/bundle/resources/postgres_branches/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/replace_existing	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/update_protected	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/synced_database_tables/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestFilerWorkspaceNotebook	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
🔄	TestFilerWorkspaceNotebook/sqlNb.sql	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R

Top 4 slowest tests (at least 2 minutes):

duration	env	testname
2:47	azure windows	TestAccept
2:38	azure-ucws windows	TestAccept
2:26	aws-ucws windows	TestAccept
2:24	gcp windows	TestAccept

shreyas-goenka · 2026-06-01T17:32:44Z

+
+	// The server validates that versionID equals last_version_id + 1 and returns
+	// ABORTED otherwise (e.g. a concurrent deploy already created this version).
+	version, versionErr := svc.CreateVersion(ctx, sdkbundle.CreateVersionRequest{


this will not work well when the plan is serialized and potentially outdated (because we do not use serial here)

Will be fixed in a followup.

shreyas-goenka · 2026-06-15T11:40:24Z

We can remove the traditional file based lock in a followup. Not necessary for now / preview.

andrewnester · 2026-06-15T12:06:12Z

+		return fmt.Errorf("failed to parse version ID %q: %w", versionID, err)
+	}
+	r.versionNum = versionNum
+	r.stopHeartbeat = startHeartbeat(ctx, r.svc, r.deploymentID, versionID)


We call CreateVersion twice: in deploy and destroy and seem to start heatbeat twice, shall we have only 1 instance of heartbeat?

Can you clarify? Those are independent code paths and both need a heartbeat right?

Ah right, my bad, indeed a separate processes

andrewnester · 2026-06-16T12:35:45Z

 		bundle.ApplyContext(ctx, b, lock.Release(lock.GoalDeploy))
 	}()
+	if err := recorder.CreateVersion(ctx); err != nil {


When we remove file lock later on, how do we ensure that there's no race condition creating multiple versions? We don't seem to do any synchronisation / locking here now, is it part of follow up?

The CreateVersion implicitly has locking semantics. Only one client can have a "live" version that is in progress at a time.

The server-side version counter is the synchronization: CreateVersion only succeeds when the requested version is last_version_id + 1, otherwise it returns ABORTED (409). So even without the file lock, two concurrent deploys racing to create the next version would have one win and the other get ABORTED. Surfacing that as a clean user-facing error (retry/serial handling) is the follow-up I noted.

Got it, make sense. Could you add an acceptance test for it though to make sure this behaviour is recorded?

pietern · 2026-06-17T09:04:51Z

+	if db.Data.Lineage == "" {
+		db.Data.Lineage = uuid.New().String()
+	}
+	return db.Data.Lineage


This is initialized in Open() below.

For direct yes. In case of DMS this is initialized at pla n time here instead. I did not want to touch direct deployment code paths.

We now have a common function to initilaize the lineage. (GetOrInitializeLineage). I confirmed that this is stored in WAL.

shreyas-goenka · 2026-06-24T09:35:18Z

+		if parseErr != nil {
+			return "", fmt.Errorf("failed to parse last_version_id %q: %w", dep.LastVersionId, parseErr)
+		}
+		versionID = strconv.FormatInt(lastVersion+1, 10)


We'll need to update this logic to also consider the existing serial numbers for migration scenarios. That'll be a followup.

denik · 2026-06-29T08:43:15Z

+	// Set up DMS recording of this destroy as a version. The version is not
+	// created until the destroy is approved (below), so a cancelled destroy
+	// records nothing; the deferred CompleteVersion is a no-op until then.
+	recorder := newDeploymentRecorder(ctx, b, engine, dms.VersionTypeDestroy)


Serial was already fixed when we created the plan, we should read it from there, not from the backend.

For clarity, we should also avoid storing serial/lineage on recorder object and just pass it to CreateVersion() directly from the plan, so that we have one source of truth for this info.

Good callout, done. We use serial from plan. In a followup I'll add support for storing and reading previous_version_id from plan as well since that requires a SDK version updatE (to get previous_version_id)

shreyas-goenka · 2026-06-29T09:32:13Z

+		// Record the DMS version now that the plan is approved (a cancelled deploy
+		// records nothing). The deployment lineage and the version's serial both
+		// come from the plan; CompleteVersion below finalizes this same version.
+		if err := recorder.CreateVersion(ctx, plan.Lineage, plan.Serial); err != nil {


in a followup - we'll also store and set previous serial in plan - to ensure serializability.

Records each approved deploy/destroy as a version with the Deployment Metadata Service (DMS), gated by experimental.record_deployment_history and the direct engine. The version is created only after the plan is approved — a cancelled or declined deploy/destroy records nothing, so there are no empty/abandoned versions for operations that never ran. - libs/dms: Recorder with CreateVersion / CompleteVersion. The deployment ID is the state lineage (from GetOrInitLineage), so a bundle deployment maps one-to-one to a DMS deployment record. GetDeployment first, CreateDeployment only when missing, then create the next version; heartbeat keeps the version's lease alive; CompleteVersion records success/failure and, for destroy, deletes the deployment record on success. Independent of bundle/lock. - phases: newDeploymentRecorder builds the recorder from the bundle (nil unless the flag is set and the engine is direct); deploy/destroy create the version inside the approved branch (after UpgradeToWrite, so the lineage is already in the WAL) and complete it in the deferred lock release. - libs/testserver: in-memory DMS handlers under /api/2.0/bundle/... - acceptance/bundle/dms: deploy/redeploy/destroy record versions and hold the file lock; redeploy after deleting .databricks recovers the lineage from remote state; enabling the flag after a plain deploy creates a new deployment; a declined destroy records no version and does not delete the deployment. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>

shreyas-goenka · 2026-06-29T11:40:05Z

+// Package dms records bundle deployments as versions with the Deployment
+// Metadata Service (DMS).
+//
+// It is intentionally independent of the deployment lock: a Recorder does not


in a followup we can deprecate the file based lock. Out of scope for now.

shreyas-goenka temporarily deployed to test-trigger-is May 31, 2026 21:29 — with GitHub Actions Inactive

shreyas-goenka changed the title ~~bundle/deploy/lock: add DMS-backed DeploymentManager implementation~~ bundle/deploy/lock: add DMS-backed DeploymentManager implementation using SDK bundle client May 31, 2026

shreyas-goenka temporarily deployed to test-trigger-is May 31, 2026 21:41 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 98dc0c7 to 4a4382f Compare June 1, 2026 12:33

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 12:34 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 4a4382f to de9adfd Compare June 1, 2026 12:35

shreyas-goenka mentioned this pull request Jun 1, 2026

bundle/deploy/lock: wire DMS into NewDeploymentLock via experimental.record_deployment_history #5395

Closed

5 tasks

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 12:36 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from de9adfd to bb16fcd Compare June 1, 2026 15:11

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:12 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from bb16fcd to bdc7ba2 Compare June 1, 2026 15:16

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:17 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from bdc7ba2 to 1e8ba45 Compare June 1, 2026 15:23

shreyas-goenka changed the title ~~bundle/deploy/lock: add DMS-backed DeploymentManager implementation using SDK bundle client~~ bundle/deploy/lock: record deployment history in DMS behind experimental.record_deployment_history Jun 1, 2026

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:23 — with GitHub Actions Inactive

shreyas-goenka mentioned this pull request Jun 1, 2026

Enable reading deployment state from DMS #5355

Open

shreyas-goenka force-pushed the shreyas-goenka/bundle-lock-abstraction branch from ff910cd to ee860e8 Compare June 1, 2026 15:31

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 1e8ba45 to 76bf017 Compare June 1, 2026 15:31

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:32 — with GitHub Actions Inactive

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:38 — with GitHub Actions Inactive

shreyas-goenka requested a review from denik June 1, 2026 17:02

shreyas-goenka commented Jun 1, 2026

View reviewed changes

shreyas-goenka requested a review from pietern June 15, 2026 11:39

shreyas-goenka assigned andrewnester Jun 15, 2026

shreyas-goenka requested a review from andrewnester June 15, 2026 11:39

shreyas-goenka unassigned andrewnester Jun 15, 2026

andrewnester reviewed Jun 15, 2026

View reviewed changes

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 607bdd0 to 98f4444 Compare June 15, 2026 13:40

shreyas-goenka temporarily deployed to test-trigger-is June 15, 2026 13:41 — with GitHub Actions Inactive

shreyas-goenka requested a review from andrewnester June 16, 2026 11:42

andrewnester approved these changes Jun 16, 2026

View reviewed changes

andrewnester reviewed Jun 16, 2026

View reviewed changes

andrewnester self-requested a review June 16, 2026 12:36

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 98f4444 to 87c99e8 Compare June 17, 2026 00:27

shreyas-goenka temporarily deployed to test-trigger-is June 17, 2026 00:28 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 87c99e8 to 9a7feb1 Compare June 17, 2026 00:28

shreyas-goenka temporarily deployed to test-trigger-is June 17, 2026 00:29 — with GitHub Actions Inactive

andrewnester approved these changes Jun 17, 2026

View reviewed changes

pietern requested changes Jun 17, 2026

View reviewed changes

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 9a7feb1 to 35f5a64 Compare June 19, 2026 07:43

shreyas-goenka mentioned this pull request Jun 19, 2026

bundle/direct/dstate: add GetOrInitLineage, the single lineage init point #5667

Merged

shreyas-goenka commented Jun 24, 2026

View reviewed changes

denik reviewed Jun 29, 2026

View reviewed changes

shreyas-goenka commented Jun 29, 2026

View reviewed changes

Uh oh!

Conversation

shreyas-goenka commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

shreyas-goenka Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shreyas-goenka commented May 31, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented May 31, 2026 •

edited

Loading

shreyas-goenka Jun 1, 2026 •

edited

Loading

shreyas-goenka Jun 25, 2026 •

edited

Loading

shreyas-goenka Jun 29, 2026 •

edited

Loading